Skip to content

feat: Swap node fetch with impit - #4

Open
trietng wants to merge 1 commit into
lnreader:mainfrom
trietng:feat/impit
Open

trietng wants to merge 1 commit into
lnreader:mainfrom
trietng:feat/impit

Conversation

@trietng

@trietng trietng commented Nov 19, 2026

Copy link
Copy Markdown

Summary

Replace Node.js's built-in globalThis.fetch (undici) with impit in @lnreader/plugin-runtime so outbound requests use Chrome's TLS/HTTP2 fingerprint (chrome151) and context-appropriate browser headers by default.

Changes

  • Added createImpitFetch() using new Impit({ browser: 'chrome151' }) as HttpClient's default fetchImpl (preserving options.fetch injection for tests).
  • Removed the static DEFAULT_HEADERS (Accept: */*, Sec-Fetch-Mode: cors) from HttpClient so impit's Chrome navigation defaults are used for page requests.
  • Updated DEFAULT_USER_AGENT to Chrome 151.0.0.0 (Windows) to match impit's chrome151 profile.
  • Added adaptHeaders() to adjust context-dependent headers (Referer, Sec-Fetch-Site, Sec-Fetch-Mode, Sec-Fetch-Dest, Sec-Fetch-User, Upgrade-Insecure-Requests) for navigation (GET), image (accept: image/*), and XHR/CORS (POST, X-Requested-With, Origin, Content-Type, non-HTML Accept) requests, and to strip sec-ch-ua* when a custom User-Agent overrides DEFAULT_USER_AGENT.

Note: With the introduction of impit as a dependency, it requires regular update as the library bump its Chrome implementation. Authentication is kept as is for logins.

Verification

  • pnpm test packages/plugin-runtime/test/client.test.ts packages/plugin-runtime/test/cookies.test.ts packages/cli-browser/test/authenticate.test.ts: client.test.ts (7 tests) and cookies.test.ts (3 tests) passed (authenticate.test.ts skipped without LNREADER_TEST_CHROMIUM).
  • Verified login session cookie capture/replay (authenticate → CookieStore → HttpClient/impit) against a local HTTP server.
  • Verified Royal Road cover image fetch (200 image/webp) without auth cookies.

Live Plugin Smoke Tests (lnreader plugins test <plugin>)

Plugin Load Popular Search Novel Chapter Notes
royalroad (2.3.4) ✔ ✔ ✔ ✔ ✔ 109 chapters, 43,344 chars
scribblehub (1.1.0) ✔ ✔ ✔ ✔ ✔ 554 chapters, 9,223 chars
webnovel (1.1.0) ✔ ✔ ✔ ✔ ✔ 3,233 chapters, 7,967 chars
wuxiaworld (0.6.0) ✔ ✔ ✔ ✔ ✔ 357 chapters, 8,035 chars
novelbin (3.0.1) ✔ ✔ ✔ ✔ ✔ 44 chapters, 3,726 chars
novelupdates (0.9.11) ✔ ✔ ✔ ✖ - parseNovel hit Cloudflare browser check
novelnice (2.4.0) ✔ ✖ ✖ - - popularNovels / searchNovels hit Cloudflare browser check
lightnovelworld (1.0.2) ✔ ✖ ✖ - - popularNovels / searchNovels returned no results

I'm not sure what to do to solve the 3 failed checks (there would probably be more, can't test all of them unless I want my IP blocked), but at least some of them should be working for now.

@rajarsheechatterjee rajarsheechatterjee left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work: TLS impersonation is a clear improvement, and build, typecheck, lint, tests and pack-check all pass locally. Two blockers: the hard native dependency (comments on impit.ts:1 / errors.ts:1) and the implicit 30 s timeout (impit.ts:91). The other two are minor. CI hasn't run on this PR yet, so a maintainer needs to approve the workflow.


Generated by Claude Code

@@ -0,0 +1,99 @@
import { Impit, type RequestInit as ImpitRequestInit } from 'impit';

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

impit is a native module with prebuilt binaries for only 8 targets: macOS, Windows, and Linux glibc/musl, each on x64 and arm64. On any other platform (Termux/Android, FreeBSD, 32-bit ARM), this top-level import makes the whole CLI crash at startup. I checked by hiding the Linux binary, and even lnreader --help failed with impit's native binding error. Could you load it lazily inside createImpitFetch (await import('impit')) and fall back to globalThis.fetch if the import throws?


Generated by Claude Code

@@ -1,3 +1,4 @@
import impit from 'impit';

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This loads the native module whenever errors.ts is imported, and nearly everything imports it. Comparing by name avoids that, e.g. e instanceof Error && /^(TransportError|ConnectError|…)$/.test(e.name), or a small isImpitTransportError helper in net/impit.ts that doesn't need the module loaded.


Generated by Claude Code

}

export function createImpitFetch(): FetchLike {
const impit = new Impit({ browser: 'chrome151' });

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

impit has a default 30 s timeout that covers the whole request, body included. Undici had no overall limit. A slow server that sends headers and then streams for more than 30 s now fails during res.text()/arrayBuffer(). That error is a plain Error rather than a TransportError, so it isn't retried or reported as NETWORK. Suggest new Impit({ browser: 'chrome151', timeout: null }), or a deliberately chosen value.


Generated by Claude Code


if (!headers.has('referer')) {
try {
headers.set('referer', `${new URL(url).origin}/`);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because Referer defaults to the target's own origin before secFetchSite runs, Sec-Fetch-Site is nearly always same-origin. The 'none' branch (line 19) never runs, and calls to other hosts send a self-referer. That's also recomputed on each redirect hop. Maybe work out Sec-Fetch-Site first and only add Referer when the caller didn't supply one?


Generated by Claude Code

headers.has('origin') ||
headers.has('content-type') ||
headers.has('authorization') ||
(accept !== null && !accept.includes('text/html')));

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: A GET with no Accept header falls through to navigation headers (text/html, Sec-Fetch-Mode: navigate). Before this PR it sent */* with cors. That's fine for pages, but plugins calling JSON APIs without setting Accept will now look like page loads. It's worth keeping in mind if any API plugins start acting differently.


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants